The PCI-X (peripheral components interface X) bus addendum is an extension of PCI bus specification. It is a data transfer protocol commonly used in high-end storage servers to transfer data to and from microprocessor and peripherals. The peripheral the subject of this publication is a disk adapter card attached to disk storage. PCI-X then is compatible with the PCI 64 bit 66 Mhz spec but also is able to handle much higher clock rate of 133, 266 and 533 providing a peak bandwidth of 4.3 Gbytes per second. A typical transfer size is 4 Kbytes of data at a rate of up to 100 times a second. The PCI-X card form-factor; pin-outs; connector; bus widths; and protocols are the same as their equivalents in the PCI specification.
The basic architecture in which a storage server connects to a PCI-X bus and system memory via a disk adapter card bus is shown in FIG. 1. The disk adapter card is an intermediary between server memory and attached disk storage, allowing data transfer between the two.
A schematic diagram of the disk adapter card is shown in FIG. 2. The disk adapter card contains a PCI-X interface device; SDRAM DIMM buffer memory; disk interface devices and an SDRAM Bus Arbiter. The disk interface device uses any standard protocol such as Fibre channel, SSA or SCSI. The buffer memory is typically SDRAM DIMM technology as this provides a suitably fast, inexpensive and dense medium for such adapter applications. The SDRAM can be accessed by: a single PCI-X interface device allowing data transfer between the PCI-X bus and the SDRAM; or one or more devices allowing the transfer of data between SDRAM and disk storage. Each device can connect to its own separate cluster of storage using the chosen disk interface protocol.
The SDRAM bus arbiter device decides which of the disk interface devices may access the SDRAM at any one time. Note that in some adapter designs, one or more of the disk interface devices may be amalgamated into one disk interface device chip but for the purposes of describing the embodiment they are shown separated.
The function of the SDRAM is to act as a buffer to compensate for any data transfer speed differentials between the PCI-X bus and the disk interface protocol. A typical DIMM bus width is 8 bytes and hence data is organised and addressed as 8-byte “words”.
A problem with the data write to disk is caused by the generation of an error-correcting code or ECC for each 8-byte word written into SDRAM. The ECC acts to provide some data error detection and correction for the data write, it is generated using the entire 8-byte word. Whenever a complete 8-byte word is to be written, a new ECC is generated from it. However, if only a part of the 8-byte word is to be written, the PCI-X device performing the buffer write must do the following: read in the entire word; modify the bytes that are to be updated with their new values; regenerate the ECC over the entire word and finally; and write the word back to SDRAM.
Such an operation is commonly known as a Read-Modify-Write. It is necessary because an ECC needs to be generated for a whole word corresponding to that stored in the SDRAM rather than just a part of word written to the SDRAM. This is not an issue when a full word is being written to the SDRAM but only when a partial word is to be written. Therefore it is necessary to acquire the whole word from SDRAM after a partial word is written.
One reason for partial word writes to occur is when the number of bytes stored in a sector of a disk is not a multiple of 8 bytes. Each sector will contain a certain sized payload of data, typically 512 or 524 bytes. This embodiment relates to sectors with 524 payloads which are not multiples of 8 bytes. Each payload may include further data checking meta data such as LRC (Longitudinal Redundancy Checking), CRC (Cyclic Redundancy Checking) or a sequence number that increments with every sector transferred. This meta data is related to the data itself as distinct from the ECC which is related to the checking of the transfer of data from the PCI-X bus to the disk. SDRAM data is organised on 8-byte address boundaries and each sector on the disk is stored on an 8-byte boundary so that it is easily addressable by each of the disk interface devices on the disk adapter card. Therefore to every 524 byte sector stored in SDRAM a pad of 4 bytes (see FIG. 5A) is added by the disk interface device which is storing the data (whether it be the PCI-X device or one of the disk interface devices), so that the following sector is also stored on an 8-byte boundary. Moreover, as each sector with padding is physically 528 bytes in size, each sector can be stored at a 528-byte aligned boundary.
U.S. Pat. No. 6,546,447 (Buckland) discloses a method and apparatus for dynamic PCI combining for PCI bridges. Buckland deals with preventing a memory controller from performing read-modify-writes (RMWs) where the data is being written from the PCI adapter to the system memory. Such RMWs can result in performance degradation when data is being transferred from system memory. Data in system memory is organised as a contiguous block of data. The PCI bridge in Buckland, acting as a target on the PCI bus, takes separate write bursts destined for contiguous addresses in system memory and buffers them together. When it has enough data to fill an integer number of cache lines of system memory it sends the data out in one burst to prevent the read-modify-write.
However Buckland only deals with the case where the PCI adapter is writing data to the system memory. It does not deal with the case where data is read from system memory, for example, by a PCI adapter and transferred into a local buffer on the adapter. Also, in Buckland, system memory is organised as a contiguous block of data so it does not need to take sector organisation into account and it does not need to deal with the case where data in system memory is organised differently from how it is organised in the local buffer.